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HYPOELLIPTIC HEAT KERNELS ON NILPOTENT LIE 

GROUPS 

MALVA ASAADt AND MARIA GORDINAt 


Abstract. The starting point of our analysis is an old idea of writing an 
eigenfunction expansion for a heat kernel considered in the case of a hypoel- 
liptic heat kernel on a nilpotent Lie group G. One of the ingredients of this 
approach is the generalized Fourier transform. The formula one gets using this 
approach is explicit as long as we can find all unitary irreducible representa¬ 
tions of G. In the current paper we consider an n-step nilpotent Lie group G n 
as an illustration of this techinique. First we apply Kirillov’s orbit method to 
describe these representations for G n • This allows us to write the correspond¬ 
ing hypoelliptic heat kernel using an integral formula over a Euclidean space. 
As an application, we describe a short-time behaviour of the hypoelliptic heat 
kernel in our case. 
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1. Introduction 


The subject of this paper is a non-commutative variation on the classical idea 
of finding an eigenfunction expansion of the heat kernel. We refer to j^] for an 
excellent review of the subject and its connections to heat kernel estimates on Rie- 
mannian manifolds. The heat kernel we consider is the hypoelliptic heat kernel for 
a sub-Laplacian on a nilpotent Lie group equipped with a natural sub-Riemannian 
structure. Recall that this setting provides a classical model for approximating of 
hypoelliptic operators first introduced by L. Rothschild and E. Stein in ps| |. 

Our goal is to describe such a hypoelliptic heat kernel in a way that allows to 
study further its properties such as short and long time behavior of such kernels. 
An eigenfunction expansion for the hypoelliptic heat kernel on SU (2) has been 
studied in [j|] , and a more abstract expansion has been presented in jfj]. The latter 
is based on a generalized Fourier transform (GFT) which informally speaking can 
be viewed as a generalized eigenfunction expansion. This requires a complicated 
machinery such as direct integrals, therefore applying it to an object such as an 
unbounded operator is not possible in general. This fundamental difficulty has 
not been addressed in Q, but we make it precise in the case of a connected simply 
connected nilpotent Lie group G. We also want to point out that even though there 
are choices to be made for an appropriate measure on a general sub-Riemannian 
manifold, the use of the GFT forces one to use a Haar measure. This complicated 
interplay between algebraic and sub-Riemannian structure in the case of Lie groups 
still needs more study. 

The main ingredient in our analysis is Kirillov’s orbit method. This technique al¬ 
lows us to describe explicitly the unitary dual of G, that is, the space of equivalency 
classes of ir reduc ible unitary representations of G. We address the domain issue 
in Theorem 3.1C which does not assume that G is nilpotent. But the description 
using the orbit method reduces this abstract descriptio n to f unction spaces over a 
Euclidean space as can be seen in the proof of Theorem 3.11. This is our main re¬ 


sult, and the explicit formula for the hypoelliptic heat kernel on a nilpotent group 
in this theorem is the result of applying the GFT to the hypoelliptic Laplacian 
on G, and then using the inversion formula for this transform. In particular, this 
formula uses a heat kernel for a certain differential operator on a Euclidean space. 
This operator is a second order differential operator with polynomial coefficients 
and a polynomial potential. We present the ingredients of both Kirillov’s orbit 
method and this Schrodinger-like operator in the case of the Heisenberg group and 
an n-step nilpotent group. In these cases this operator is the Schrodinger operator 
with a quadratic potential on a certain ! iV . For related results we refer to p2| , |23| . 
The dimension of this Euclidean space and the degree of the potential depend on 
the structure of G in terms of the orbit method. This connection with the heat 
kernel for such operators provides a way to prove bounds and functional inequalities 
for the hypoelliptic heat kernel. We only mention a simple estimate in the paper, 
but we expect that this approach will be used in the future. Moreover, it would 
be interesting to see if the short time estimates one gets from this analysis can be 
formulated in terms of the sub-Riemannian distance. Our main new example is an 
n-step nilpotent group, though we also mention previously studied examples, such 
as the Heisenberg group. 
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The paper is organized as follows. In Section g we review Kirillov’s orbit method 
for nilpotent groups and illustrate it by describing explicitly the irreducible repre¬ 
sentations of the n-step nilpotent Lie group of the growth vector (2,3, ...,n + 1), 
G r i+i. This turns out to be an inductive generalization of the computations car¬ 
ried out in [^5} for the Heisenberg group. In Section [| we recall basic definitions 
from sub-Riemannian geometry, in particular for left-invariant structures on Lie 
groups. In Section we use the generalized Fourier transform and its properties 
to describe the hypo-elliptic heat kernel on a nilpotent group. Finally in Section 
[|, we apply this technique to the group G n +1 . In Section we use the formula for 
the hypo-elliptic kernel to study the short-time behaviour of heat kernels. In that 
we are motivated by the approach introduced by Seguin and Mansouri |^o|. Note 
that we avoid introducing assumptions as in ]3(], p.3904], as we can use the explicit 
expressions we derive using the orbit method, and thus avoid ambiguities in their 
assumptions. 

There are a number of papers related to the subject of this paper, and below we 
mention just a few. Besides the ones we already referred to earlier such as recent 
results in ill [30| , w e would like to mention earlier explicit hypoelliptic heat kernel 
expressions in Hi- Our approach is different as we use Kirillov’s orbit method 
to represent a hypoelliptic heat kernel on a nilpotent group by an integral formula 
over a Euclidean space. This is what makes it different from using analysis on the 
group itself as is done in Jlo|,[l9) . We leave out a vast literature on various estimates 
on hypoelliptic heat kernels. 


2. Kirillov’s orbit method 

Kirillov’s theory gives an explicit description of irreducible unitary representa¬ 
tions of a nilpotent Lie group. Throughout this paper we assume that G is con¬ 
nected and simply connected. The main ingredient of Kirillov’s theory is the set 
of the orbits of the coadjoint action. Namely, the coadjoint map allows to describe 
all irreducible unitary representations of G: the set of equivalency classes of all 
irreducible unitary representations of G are naturally parameterized by the orbits 
of g* under the coadjoint action. In this section, we first review Kirillov’s orbit 
theory ^5|, || Section 3.1], and then apply it to an n-step nilpotent Lie group 
G ra _|_i which allows us to describe explicitly all irreducible representations of G„+i. 

Recall that a Lie group G acts on its Lie algebra g by the adjoint action 


Ad g X := gXg 1 ,g GX G g, 

and G acts on the dual space g* by the coadjoint action as follows 


Ad* g l(X) :=l(g- 1 Xg),gGG,lGg*,X Gg. 

Finally, the orbit of l in g* under the coadjoint action Ad* is denoted by Oi. 
Definition 2.1. For an element l G g* we define its radical by 
radi := {Y G g : l[X, Y] = 0 for all l£g}. 

An equivalent description of the radical is given by 


radi = {Y G g : ((ady)* l ) ( X ) = 0 for all If eg}, 
and radi is a subalgebra of g. 

The next step in Kirillov’s method is to find a polarizing subalgebra. 
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Definition 2.2. An ideal m C g is called a polarizing (or maximal subordinate) 
subalgebra for l if it satisfies 

(1) radi C m; 

(2) dim m = d™™rf,+dim B ; 

(3) l ([m, m]) = 0. 

The last property means that this subalgebra is subordinate for l , and the second 
one indicates that the subalgebra is of maximal dimension. The first property is 
actually a result proven first by Pukanszky in (2^, p. 157]. Recall that for a 
nilpotent Lie algebra such a subalgebra always exists by || Theorem 1.3.3 ]. 

Remark 2.3. As p.30] says for each l £ g* its radical is uniquely determined, 
but there might be more than one polarizing subalgebra for l. When one wants to 
avoid this ambiguity, one can use Vergne’s construction in which gives a 

canonical way to choose a polarizing subalgebra for l £ g* and a strong Mal’cev 
basis in g. 

2.1. An n-step nilpotent Lie group G n +i- We would like to illustrate how to 
find these objects by considering an n-step nilpotent Lie group G n +\ also known 
as a thread-like group. Let g n +i be an (n + l)-dimensional Lie algebra generated 
by X, Yj,..., Y n satisfying 

[X, Yj] = Y i+ 1 , [X, Y n ] = 0, and [Yi,Yj] = 0 for all i,j. 

This is an n-step nilpotent Lie algebra. A realization of Q n +i as a matrix algebra 
is obtained by letting W = aX + ]G" =1 bjYj corresponding to the (n + 1) x (n + 1) 
matrix of g n +i 


(0 

a 

0 

0 •• 

• 0 

bn \ 

0 

0 

a 

0 •• 

• 0 

bn— 1 

0 

0 

0 

a ■ 

• 0 

bn— 1 

0 

0 

0 

0 •• 

a 

b 2 

0 

0 

0 

0 

■ 0 

bi 

\0 

0 

0 

0 •• 

• 0 

o / 


Let G n +1 be a connected, simply connected nilpotent Lie group, with the Lie algebra 
g„+i, then the exponential map exp : g„ +1 -A G„+i is an analytic diffeomorphism. 
Moreover, if 


then 


n 

W = aX + biYi £ 0n+l? 
2=1 


w = exp ( aX + biYi ) € G n +1 
2=1 
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can be written as 


( 2 . 1 ) 


where 

( 2 . 2 ) 


0 1 a | 

0 0 1a 


Zn \ 
Z n -1 
Zn -2 


0 0 0 0 ■ • • z\ 

V 0 0 0 0 ••• 1 / 


j- 1 

z i := E ' 

fc =0 


a k bj-k 

(fc + 1)! 


j = 1, 


2.2. Coadjoint action, coadjoint orbits and polarizing subalgebras for G n . 

Suppose 


W = aX + ^ biYi e 


0n+l •> 


i— 1 


w = exp(a;X + ^ i/iYi) £ G n +i, 
then the adjoint action is given by 




Ad w W = wWw = 


/ 

0 

a 

0 

0 

... o 


\ 


0 

0 

a 

0 

... o 

£ra—1 



0 

0 

0 

a 

... o 

£n—1 



0 

0 

0 

0 

a 

& 



0 

0 

0 

0 

... o 

£l 


\ 

0 

0 

0 

0 

... 0 

0 

/ 


where 


^ x k-1 

= bj + 2J - «%-*}• 


fc=l 


Finally we can use the basis { X , Yi,Y n } of g n +i and the dual basis {X*, Y x *, Y*} 
of g* +1 , to see that the co-adjoint action is given by 

/ n / i-1 

Ad* w (l)(W) = l(Ad w -iW) = aia + J2^\Y,^ k+1 y^ : 


Jfe-1 


i —2 \k =1 


ft! 


E U-XX-i) 

3= 1 V k=j 


k—j 


r k~3 


(k~ j)\ 


for any l = aX* + £” = i ft*? G 

For to G N we define the polynomial function 


/i(*;B m ) :=]T(-l) fe ^ 

k=j 


Pk (Pm-1 - X) k 3 

P™ J (k — j)\ 


(2.3) 
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where i£l, B m = (/3i,/3 2 , -,Pm) £ ^ m ,Pm ^ 0. 

Observe that the orbit Oi of l = aX* + PiY* -\ -+ P m Yf^ with P m ^ 0 under 

the coadjoint action is two-dimensional if m ^ 3 and is given by 


— 0(o,B mi 0,..,0) — 


m —2 

yX* + J2 /jfo B rn) Y * + + 0 m Y£, 1 , 1 /eR 


Here we identified l £ 0 * with (a, B m , 0,0). In case of m = 2 then the orbit of 
l = aX* + PiY* + P 2 Y 2 is again two-dimensional and has the following form: 

= Y)( a ,p i,/3 2 ,o,...,o) = {vX* + qYf + P2Y2 ,p , q G R} • 

and in this case 

radi = {62^2 + • • • + b n Y n , bi £ K} 

Finally, in the case where m = 1 then the orbit of l = aX* + fiiY* is a zero- 
dimensional orbit (point orbits) 


Oi = CW ,0 . 0 ) = {aX* + PlY*} . 


and in this case radi = g. 


Proposition 2.4. For any l £ g* +1 and to ^ 3 


radi = 

m —3 


E 

j=i 


fj+l (0; B m ) , 
Pm 


Y m —i ) -(- b m —2Y rn —2 + b m Y m + • • • + b n Y n 


for some B m , where /j+ i(0;B m ) is defined by (2.3). 


Proof. Note that the definition of the radical is independent of the representative 
in the orbit, therefore it is enough to compute the radical for elements 

m —2 

(2-4) 1= J2 fj(0;B m )Y; + p m Y£. 

j =1 

An arbitrary element of g n +i is of the form 


and therefore 


n 

Z = aX + J2 bfYi 
2=1 


[X, Z\ — 61 I 2 + • • • + b n -iY n , 
[dm— 1 , Z] = nd m . 


Thus 


radi = {Z £ g„+i: l[X, Z) = l[Yi, Z] 


l[Y n ,Z} = 0}, 
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and so we get the relations 


fim® — Oj 

/3 3 &2 = 0 if m = 3, 


771 —2 

Pmbm-l + ^2 = 0 if 771 ^ 4. 

j=2 

Recall that /3 m ^ 0, and therefore it is enough to consider two cases as follows. 
Case 1: m = 3. In this case a = 6 2 = 0 and the radical is 


radi = Span{Pi,P 3 ,P 4 , ...,Y n } . 
Case 2: m > 4. In this case we get a = 0 and 

771 —2 , 

6 ro _ 1 = -^ 


i=2 


At 


□ 


Now we can describe polarizing algebras for g„+i explicitly. 

Proposition 2.5. Suppose l = al* + /3iPi* H-h /3 m P r * G 0 ^+i with /3 m ^ 0 /or 

some m ^ 3 ; t/ien a polarizing algebra for l is unique, and is given by 


m = Span {Pi,..., P n }. 


Proof. By Proposition 2.4 we see that dim radi = n— 1 and since dimg„ + i = n + 1, 


we have dimm = n. This means that to find a polarizing subalgebra it is enough to 
add one element Y € 0 n +i to radi such that P is linearly independent of radi an d 
the ideal generated by radi an d P is not the whole algebra 0 „+i- It can be easily 
seen that such an m is unique and it is equal to 


m = Span {Pi,..., Y n j 


as long as m ^ 3 in (PJ). 


□ 


For the case m = 2 the orbit is again two-dimensional, but the polarizing subalge¬ 
bra is not unique. For example, the following two ideals are polarizing subalgebras 

mi = Span {Pi, P 2 , ...,P„} , or 
m 2 = Span{X, P 2 , ...,P„} . 

Finally, for m = 1 the zero-dimensional orbits are one-point orbits, namely, 

,0.0) = {L,/ 3 i ,0 .0} , 

in which case radi = m = 0 rl+ i. 


2.3. Induced representations for nilpotent Lie groups. To apply Kirillov’s 
orbit method to the representation theory of nilpotent groups we first recall some 
basic facts about induced representations which can be found in [||,|l 2 | among many 
other sources. 

Definition 2.6. The set G of equivalence classes of irreducible unitary represen¬ 
tations of a locally compact group G is called the (unitary) dual space of G. 
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If M is a closed connected subgroup of G , we can use the induction procedure 
to describe representations of G via representations of M. Suppose (x, %x) i s a 
representation of M. Then (x, 7 i x ) yields a natural representation ir = Ind(M f 
G, x) of G in a new Hilbert space 7 i^. We start by constructing the representation 
Hilbert space Tt* . This space is defined as a Hilbert space of equivalence classes of 
Borel measurable vector-valued functions / : G —>• 7 i x such that 


f(mg) = xM 1 f(g), me M,geG, 

(2-5) J \\f{g)\\ 2 dg{g) < oo, 

G/M 

where g ( g ) is a left-invariant measure on G/M. 

Note that in our case both G and M are nilpotent and thus unimodular, and 
therefore such a measure always exists on the homogeneous space G/M (e.g. [|| 
Lemma 1.2.13]). In measure-theoretical terms the measure /i is the pushforward of 
the Haar measure dg by the quotient map 


q : G —» G/M , 

(2.6) q (g) := gM. 

For nilpotent groups the induction procedure described above is applied to a 
particular choice of the representation x, namely, for a fixed l e Q* we define the 
character of M by 


(2.7) xi (exp (X)) := exp (2ml (X)), X <E m, 


where we used the fact that exp : m — 
this case we choose the Hilbert space 7i ni „ 


M is a (global) diffeomorphism. In 
to be the Hilbert space of equivalence 


classes of Borel measurable functions / : G —> C satisfying (2.5). Note that the 
map g i-A ||/(g)|| 2 is constant on each left coset, so the integral in (2.5) exists. The 
space Tim is a completion of the space of such functions with respect to the inner 
product 


(h{a)ih{g)) ~ f {fi(g),f2(g))dg(g) 

JG/M 

which is again well-defined since (fi{g), /2Q7)) is constant on each left coset. 

Finally the induced representation 717 , m is defined by letting G act on the right 
as follows 


7i7,m(z)/(s) := f(x 1 ff), for all x e GJ e 7i m , m , 


which is a unitary operation. This is an irreducible representation of G on 7i^ l m , 
and any irreducible representation of G can be written as an induced representation 
for some l € 0 * and any polarizing subalgebra m for l by j|, p. 124], That is, for any 
l € 0 * and two polarizing subalgebras these representations are unitarily equivalent. 

By H pp. 124-125] and (|T2|, p. 159] there is an isometry between 7i x and 
L 2 (R fc , dx) for some k with respect to the Lebesgue measure dx. This allows us to 
find a representation on L 2 (R fc , dx ) which is unitarily equivalent to 7r/ m . Moreover, 
we can identify smooth vectors C°° of the representation 717 . m with the Schwartz 
space on this R fc . 
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To make these isometries more explicit in the nilpotent case we need to choose 
a weak Mal’cev basis of g passing through m, that is, a basis {X \,..., X n } of g such 
that {Xi, ..., Xk} is a basis of m, and Span{Xi, Xj} is a subalgebra of g for any 
1 ^ j ^ n. This basis allows us to have an explicit description of G/M as described 
by the following theorem which is based on j|, Theorem 1.2.12 ]. 

Theorem 2.7. Let m be a k-dimensional subalgebra of the nilpotent Lie algebra g, 
let M = exp(m) and G = exp(g), and let {Xi, ...,X n } be a weak Mal’cev basis for 
g through m. Define (f : R ra-fc — > G/M by 

(2.8) <j)(x 1 ,...,x n - k ) := exp(xiX k+1 ) ■ ■ ■ exp(x n - k X n ) ■ M. 

Then <f> is an analytic dijfeomorphism which is also a measure space isomorphism 
from (R ra-fc , dx), where dx is the Lebesgue measure, onto ( G/M,p), where p is the 
left-invariant measure on G/M. 

Denote 


DTI —fc 


(2.9) 


7 : K" - —> G, 

7 (x \,..., x n -k) := exp (xiX k +i) ■ ... • exp (x n - k X n ), 


and recall that the quotient map q : G - 
isomorphism. Then the map <j> in Theorem 


G/M in (2.6) is a measure space 
can be written as 


Observe that the map 

( 2 . 10 ) 

is a diffeomorphism from . 


(p = q 07 . 

if : R ra-fc x M — >G, 

(x, m) 1 —> 7 (x) m 

' l ~ k x M onto G. We denote the inverse of this map by 


( 2 . 11 ) 


ip~ := (pi,p 2 ), where 
Pi : G —* M. n ~ k , 

P 2 '■ G —> M 


Using the diffeomorphism if and the fact that by Theorem 2/7 the map <f is a mea¬ 
sure space diffeomorphism from M n ~ k onto G we can induce a unitary isomorphism 


( 2 . 12 ) 

J : L 2 (R n ~ k ,dx) 


TL-n 


( jf) (7 (x), m) := Xi ( rn ) 1 / (x), for all x £ 


i—k 


m £ M, f £ L 2 


% — k 


dx). 


It is easy to see that Jf £ H VI m since it satisfies (2.5), and the inverse map is given 
by 


J - 1 :'H^ m ^L 2 (R n - k ,dx), 

( J~ x h) (x) := h (7 (®)), for all x £ R n ~ k , h £ 
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Finally, we can describe a unitary representation of G on L 2 ( R” k , dx) which is 
unitarily equivalently to 7 r^ ?m . Namely, 


(2-13) {Ui, m ( g ) /) (x) := Xl ( P 2 (7 (x) ff)) f (p 1 (7 0 ) s)) , 

for all / <E L 2 ( W l ~ k , dx),g G G.a.e.x G 
The unitary equivalency of ir^ m and £/z jVn can be shown by using the map J 

J 7^1,mJ — 

Note that this unitary representation Ui tXn depends on the choice of the weak 
Mal’cev basis, but any of these choices gives rise to unitarily equivalent repre¬ 
sentations. 


Remark 2.8. We will abuse notation and denote the representation U^ m by ir^ m 
whenever it is clear that the representation space is L 2 (W l ~ k ,dx). 


2.4. Irreducible unitary representations of the group G n + 1 . Now we apply 
the induction procedure described in Section 2.3 to the group G „+1 introduced in 
Section 2.1. 

First we identify a point (b, a) G R " +1 with a point in the group G „+± by 


(b, a) := exp (b n Y n + -|-b h Yi + aX). 


Theorem 2.9. Let G n +1 be the n-step nilpotent Lie group described in Section 2.1, 
then all (nonequivalent) unitary irreducible representations of G n +1 are as follows. 
(1) The infinite dimensional representations of G n +1 on the Hilbert space = 
L 2 (R,dx) are given by 


2 mil V B k (x)Y k ) 

ni(h,a)f{x) := e '/(x + a), 

k 

(2.14) B k (x) := J2 Ji^Tfy xk ~^ k = 2 ’ ■ • ■.", 

for f G L 2 (R, dx), x Gl,(G 0(a,B m ,o,...,o)> 3 < m < n. 

(2) The one-dimensional unitary representations (characters) on H n = C are 
given by 

^(b, a) = 1 e O {M _ 0) 

Proof. First we describe the representations of G n + 1 corresponding to the two- 
dimensional orbits, that is, 

G(a, B m ,0,...,0) = 

jxX* + fj(y;B m )Y* + yY((_ 1 + /3 m Y^: x,y G R,m ^ 2 J 
with B m defined by (|j^) and 

m —2 

i = ^/,(0;B m )y; + / 3 m Y r ; 

t=i 
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being orbit representatives for /3 m ^ 0 for the polarizing subalgebra 


m = Span{Yi,y 2 , —,Yn} ■ 


Then for W = Yli=i b%Y% and this choice of l and m we see that as in (2.7) 

Xi (exp(WO) = 


defines a character of the subgroup M = exp(m). Using (2.10) we identify G with 

R x {exp(xJf), x € R} 


via the diffeomorphism ip. Now we can describe the action of 7 r/, m on L 2 (R, dx). 
For (b, a) = exp (b n Y n + b n _iY n _i + ... + b{Y\ + aX) we have 


( 0 ,..., 0 , x) • (b, a) = 


n —1 


y.n—1—i 


^ ^ / ■ \ I 5 ^ ^ 1 -x. ? • * • 5 ^1? 0 ] (0, . . • , 0, X H - a) , 

^ (n- 1 -i)\ J 


where Zi are defined [2.2[ In terms of ( 2.11 ) this identity can be written as 


Pi (7 (z) g) = Pi ((0, • • •, 0, x ) • (b, a)) = pi ((0, ■ • •, 0, x + a)) 
= x + a, 

P2 (7 (z) 9 ) = P2 ((0, • • ■ , 0, x ) ■ (b, a)) = 

/ ( n rf.n —1 T n-l-i \ \ 

(B n (%) ? B n — 1 {%) 5 •••5 (x)) , 


where 


B k ( x ) = y ^Zj 


r,k — i 


i= 1 


(k - i)\ 


k = 1 , 2 ,...,n. 


Identifying 'H Vl m with L 2 (R, dx) we see that 

77, m(b, a)/(x) = xi (P 2 (7 (a:) 5)) / (Pi (7 0) 3)) 

= B k (x)Y k )j^ x + a y 

Now, for the one-point orbit 0( a j 3 ^ 0 ,...,o) = {&X* + (3{Y*}, a,/3 1 € R, we have 
seen that m = g n+1 and induction from M = exp(m) is trivial. Thus 7T; = xi is 
one-dimensional and is given by 

7rz(b, a) = e 2 ™( Q “+^ b i)/ 


with H 7 r, = C. 


□ 


2.5. Generalized Fourier transform. We start by reviewing the generalized 
Fourier transform (GFT) as described in [[[ 2 ], Section 7.5]. First we assume that G 
is a separable locally compact unimodular Lie group of type I, and L P (G,C) de¬ 
notes the space of complex-valued functions on G which are square-integrable with 
respect to the Haar measure dg. Later we consider this setting in the case when G 
is an addition nilpotent. 
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Recall that we defined G to be the (unitary) dual space of G in Definition 2.6 


The structure of the dual G can be described explicitly for some classes of groups 
such as locally compact Abelian or compact groups. Another class of groups for 
which one can find an explicit description of G is of simply connected nilpotent Lie 
groups as we described in Section 2.3. 


Remark 2.10. The GFT is usually defined using the structure ofG as a measurable 
space, and therefore G should have nice properties such as being countably separated, 
since not countably separated measurable spaces are pathological. By , |7^ , Theorem 
7.6] this is equivalent to G being of type I. In particular, if G a simply-connected 
nilpotent Lie group, then by |^.[a^/ the group G is of type I. 


Once we have equipped G with the structure of a nice measurable space, we can 
define a Borcl measure P on G called the Plancherel measure. Then a possible issue 
is how to make a measurable selection of 7r^ for £ £ G. As observed in [[12], p. 230] 
this can be done if G is of type I. Moreover, if G is a simply-connected nilpotent Lie 
group, the Plancherel measure can be identified with the Lebesgue measure with a 
density on R 9 for some q using the unitary isomorphism J introduced in ( [2. 12 ). 

For (eG we will denote by 7a choice of an irreducible representation in the 
equivalency class £, and the representation Hilbert space by H n t or 


Definition 2.11. For / £ L 1 (G,C) the generalized Fourier transform (GFT) of 
/ is the map P(f) (or /) that takes each element of G to a linear operator on the 
representation space by 


(2.15) P(f) (6 = 7(0 = 7fo) := f f{g)^tg- l )dg, for P - a.e.£ G G. 

Jg 

As is known, the Fourier transform P(f), f £ L 1 (G, C) fl L 2 (G, C), is a Hilbert- 
Schmidt operator for almost all £ £ G with respect to the Plancherel measure P, 
and the map £ / (£) is a P-measurable field of operators which allows for use 

of direct integrals, and can be used to find a spectral decomposition of differential 
operators on L 2 (G, C). In particular, one can prove the Plancherel Theorem and 
the Fourier inversion formula in this abstract setting (e.g. [|l2]. Theorem 7.44]). The 
Plancherel Theorem gives rise to an extension of the GFT (which we again denote 
by F) to an isometry 


(2.16) 


P : L 2 (G, C) 


HS (U^)dP((), 


where HS (H v i) is the space of Hilbert-Schmidt operators which is a Hilbert space 
itself. In the case of G being nilpotent, the Plancherel measure and the space of 
Hilbert-Schmidt operators can be described explicitly. As it is done over Euclidean 
spaces, we will use the Fourier transform in Definition 2.11 to find a spectral de¬ 
composition for the left-invariant vector field X corresponding to any X £ g. In 
this we will use the following notation. Let T be a linear operator on L 2 (G, C), 
then we denote by T the following linear operator 


T: lKS(H^)dP(f) 


hs (n^)dP(O, 


T = FTP*, 


(2.17) 
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where JF* is the adjoint of J- 

T* : [ HS (H n e)dP(0^L 2 (G,C). 

JG 

Remark 2.12. We would like to mention here that some of the standard properties 
of the GFT are listed in |i|/. and a few of them should be interpreted with caution. 
For example, Equation (18)] gives a formal expression for the GFT of the Dirac 
mass measure, which needs an introduction of an analogue of tempered distributions. 
We note here that the theory of tempered distributions have been studied on nilpotent 
groups (not general unimodular groups), see HI# 


2.6. The GFT and the Plancherel formula for nilpotent groups. We start 
by recalling that for connected, simply connected nilpotent Lie groups the GFT 
can be described explicitly. In general for an infinite-dimensional irreducible rep¬ 
resentation the operator it ( g ), g £ G is not necessarily a trace-class operator. Let 
S(G) be the space of Schwartz functions on G as defined in || Appendix A.2], then 
for / £ S(G ) the operator n (/) defined by (2.15) is a trace-class operator. 

By Theorem 4.2.1] for th e irreducible unitary representation 7iy m on L 2 (W rl ~ k ,dx) 
identified with Ui m by (2.13) we have that for any f £ S (G) there is an integral 


kernel kf £ S I 


d n—k 


T>n—k 


) such that for h £ L 2 (]R” k , dx ) 


(/(7p,m)/i) (a 


h (y) k f (x, y) dy. 


By 1 Proposition 4.2.2] the kernel kf ( x,y) is given by 


dm. 


(2.18) k f (x,y)= [ xi (m) f (7 (x) 1 my 

JM v 

whe re x i is the character on M defined by (2/7), 7 is a map R” -fe —> G defined 
by (2.9). and dm is the Haar measure on M. This is what ter Elst and Robinson 
call a reduced kernel in ]32|, p. 481, (4)]. 

Now we turn to the Plancherel Formula. Our main goal here is to identify the 
Plancherel measure with a measure on a Euclidean space. Let {Xi,..., X n } be a 
basis for a nilpotent Lie algebra g, and let {X]",..., X*} be the dual basis for g*. We 
can use H Theorem 3.1.6, Corollary 3.1.8 ] to find a G-invariant set U of generic 
coadjoint orbits in g and two disjoint sets of indices S, T that partition {1 ,... ,n) 
in such a way that (so-called) jump indices of any l £ U is exactly S. For more 
details on jump indices etc we refer to || p. 84]. Let 


glj := Span R [X*, i £ S}, 
B t == Span R {X*, j £ T}. 


Note that the skew-symmetric form Bi(X,Y) := l([X, F]) on g has the radical 
radi , and therefore induces a non-degenerate skew-symmetric form on g/ radi. 


Definition 2.13. If l £ g* and {Ad,..., Xik} is a basis for g/radi, then the Pfaffian 
Pf (7) is defined by 

Pf (7) 2 := det B l , where 
B l (X,Y) :=l([X,Y]),X,Y £g, 

B\f := B l (Xi, Xj). 
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First we recall || Theorem 4.3.9] combined with |u|, p.374, (0.3)], where as 
before S(G ) is the space of Schwartz functions on G. 

Theorem 2.14 (The Fourier Inversion). Let {Xi ,..., X n } be a basis for a nilpotent 
Lie algebra g, and let { X*,...,X *} be the dual basis for g*. Define U,S,T and 
Pfaffian as above, then for f £ S(G), /(e) is given by an absolutely convergent 
integral 

fig) = [ Tr U m (ff- 1 ) / ( n.m)) I m)\di, 

Jun v J 

where dl is the Lebesgue measure on gif. 

Note that l £ g* can be identified with a point in a Euclidean space by using 
coordinates of l in the dual basis {X^,..., X*}. The following theorem can be 
found in Theorem 4.3.10] 

Theorem 2.15 (Plancherel Theorem). Let notation be as in the previous theorem. 
For f £ S(G) we have 

11/111= / ll/Km)||^|Pf(0|di, 

Ju nflj, 

where ||A||#s is the Hilbert-Schmidt norm of an operator A. 

Remark 2.16. Note that usually we identify U D 0^ with an open subset of a 
Euclidean space, and therefore dl is simply the Lebesgue measure. 

Definition 2.17. The measure dP := | Pi(l)\dl is the pushforward of the Plancherel 
measure on G , namely, P is a Borel measure defined by the identification G = 
g* / Ad* G. We will abuse notion and call this pushforward measure the Plancherel 
measure. 

In our setting we can explicitly identify the ingredients needed for application of 
these theorems. 

Proposition 2.18. ForG n +\ the space is isomorphic toK" -1 and the Plancherel 
measure is given by 


dP —|A„_i|dAidA2 • • • d\ n -i. 

The Plancherel identity then becomes 

11 /11 2 = / ll/(7ri)llff5|An- 1 |dA 1 dA 2 ---dA n _ 1 , 

JR"- 1 

where ni are the unitary representations described in Theorem \2.[\ and l = (Ai,..., A„_i) 
is identified with a point in M ra_1 . 

Proof. If {Y *,..., Y-f. X*} is the dual basis for g* +1 , the indices {1,..., n+ 1} can 
be partitioned as S = {2, n + 1} and T = {1,3,4,..., n} so that 

(g; +1 ) T = Span R {y n *, y;_ 2 ,..., y/}, 

and the generic orbits are of the form 
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Thus we see that 


( 0 ;+i) T nu= {Xu-iY: + A „_ 2 y „*_ 2 + • • • + XiY? :A t eR, A n _! ^ 0 } . 

The Pfaffian is a polynomial on 0 *.! such that 



Note that {Y*_ x , X*} is a basis for Q n+ \/radi. Identifying (g* +1 )T with R ” _1 and 
letting dXi ■ ■ ■ dX n -i be the Lebesgue measure, we have | Pf(Z)| = |A„_i|. If 7 iy m is 
the representation corresponding to 



the Plancherel formula becomes 



and the Plancherel measure is identified with the following measure on M" 1 


(2.19) 


dP — |A n _i|dAidA 2 • • • dX n -i . 


□ 


3. HYPOELLIPTIC HEAT EQUATION ON A SUB-RlEMANNIAN MANIFOLD 

In this section, we start by reviewing some standard definitions in sub-Riemannian 
geometry, and in particular, how a natural left-invariant sub-Riemannian structure 
on nilpotent Lie groups is constructed. 

3.1. Sub-Riemannian manifolds. Let M be an n-dimensional connected smooth 
manifold, with tangent and cotangent bundles TM and T*M respectively. 

Definition 3.1. For m ^ n, let TL be a smooth sub-bundle of TM, where each 
fiber TL q has dimension m and is equipped with an inner product g which smoothly 
varies between hirers. Then 

( 1 ) the triple (M, 7t,g) is called a sub-Riemannian manifold of rank m; 

(2) TL is called a horizontal distribution on M, and the inner product g a sub- 
Riemannian metric ; 

(3) sections of TL are called horizontal vector fields , and curves on M whose 
velocity vectors are always horizontal are called horizontal curves. 

Assumption 3.2 (Hormander’s condition). Throughout this paper we assume that 
the distribution TL satisfies Hormander’s (bracket generating) condition; that is, 
horizontal vector fields with their Lie brackets span the tangent space T p M at every 
point p € M. 

Under Hormander’s condition any two points on M can be connected by a 
horizontal curve by the Chow-Rachevski theorem. Thus there is a natural sub- 
Riemannian distance (Carnot-Caratheodory distance) on M defined as the inhmum 
over the lengths of horizontal curves connecting two points. In turn, this affords 
us the notion of a horizontal geodesic , a horizontal curve whose length (locally) 
realizes the Carnot-Caratheodory distance. 

Definition 3.3. Let (M,TL,g) be a sub-Riemannian manifold. 
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(1) Locally (H, g) can be given by assigning a set of k smooth vector fields 
spanning T~L. Suppose these vector fields can be chosen so that they are 
orthonormal with respect to g 

Up = Span{Xi(p ),... ,X m (p)},g p (Xi{p),Xj(p)) = 6ij, 

then we say that the sub-Riemannian manifold M is trivializable. 

(2) If M is analytic and the vectors {Xi,... ,X m } are analytic vector fields, 
we say the sub-Riemannian manifold (M,R, g) is analytic. 

(3) M is regular if for 

Hi :=n,n i+ 1 := + 

the dimension of Ri(p), i = 1,2,... does not depend on the point p £ M. 

3.2. Left-invariant sub-Riemannian structure on Lie groups. Let G be a 

Lie group with Lie algebra g := T e G. As usual we identify T e G with the set of 
left-invariant vector fields on G as follows. First consider smooth diffeomorphisms 
L a and R a of G, namely, the left and right translations by an element p £ G 


Lp ‘ h i—^ ph, h £ G, 

Rp \ h i —y hp, h £ G. 

Notation 3.4. For any X £ g we denote by X the unique left-invariant vector 
field such that X (e) = X, that is, for any f £ C°° ( G ) 


(3.1) 


Xf{g) := 


d_ 

dt 


f(ge iX ) 

t=o 


d 

dt 


f (Re tx g ); 

t=o 


where R p : G G is the right translation by an element p £ G 


The Lie group G can be equipped with a left-invariant Riemannian metric as 
follows. First an inner product (•, -) e on T e G = g determines a left-invariant metric 
on G by the identification dL p : g = T e G —> T p G , for any p £ G. Then we can 
define the corresponding Riemannian metric (•, •) on G by 


(3-2) (X, Y) p := ((dV0 P W - (dL P - 0 P ( Y ))e 

for any p £ G and X, Y £ g. Another way to write this is as the pull-back 
(dL p )* g = g. Conversely, if a Riemannian metric on G satisfies (T2), then it 
is called left-invariant. 

Now we can describe a similar construction in the sub-Riemannian setting. 


Definition 3.5. Let G be a Lie group with Lie algebra g, H be a distribution 
satisfying Hormander’s condition (Assumption |3.2| ), and g be a sub-Riemannian 
metric. We will say that (G, H, g) is equipped with a left-invariant sub-Riemannian 
structure if 

(1) the distribution H is left-invariant, that is, H e is a linear subspace of the 
Lie algebra g such that 

H p = LpUe for p £ G; 
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(2) the metric g is left-invariant, that is, 

g p (X,Y) = g e ((dVOp (*) > ( dL P- Op ( y )) G G,X, F e 0. 

Finally, we observe that all left-invariant sub-Riemannian manifolds are regular 
and trivializable according to |l], Remark 5, p. 2627]. 


3.3. The hypoelliptic Laplacian. As we observed in the introduction, there are 
several choices that need to be made to define a natural hypoelliptic Laplacian 
corresponding to the sub-Riemannian structure at hand. One of these choices is of 
a measure, and as is seen from JI], Remark 16] in the case of Lie groups equipped 
with a left-invariant sub-Riemannian structure the Haar measure is a natural choice: 
the Popp measure defined in |p6|| and the Hausdorff measure are both left-invariant, 
and therefore are proportional to the left Haar measure. 

Once the measure is chosen, an operator defined as a divergence of the horizontal 
gradient is the usual sum of squares operator. Note that the argument in jjll has a 
mistake, and for more detailed discussion of related issues we refer to ||T^,|lJ. But 
no matter which point of view we use, in the case of a nilpotent Lie group all these 
approaches give the same result: the sum of squares operator. 

Let (G, ?t,g) be a unimodular Lie group equipped with a left-invariant sub- 
Riemannian structure of rank k. Our goal now is to see how we can use the GFT to 
diagonalize left-invariant vector fields, and therefore the sub-Laplacian A h- This 
is really a non-commutative analogue of the Euclidean case. One of the issues we 
want to clarify in what follows is the domains of the operators involved. This is 
something that is missing in [Q, and can be made explicit in the case when G is 
nilpotent. 

Define A h to be the left-invariant second order differential operator 

m 

A ff /:=^^ 2 /, /GG C °°(G), 

i=l 


where {X i }])L 1 is an orthonormal basis of JH e i g e ). 

The operator A h is a densely defined symmetric operator on L 2 ( G,dg ), where 
dg is a right-invariant Haar measure. It has a self-adjoint extension, namely, the 
Friedrichs extension, which we will denote by the same Ah- For details we refer 
to B Section II.5]. In addition we assume that P. satisfies Hormander’s condition 


3.2, therefore the operator A h is hypoelliptic by M 


Definition 3.6. Let P t denote the heat semigroup e tAu , where A h is the self- 
adjoint (Friedrichs) extension of A h \c^>(G) t° L 2 (G,dg), where dg is a right- 
invariant Haar measure. By the left invariance of Ah and Hormander condition 
(pT^), P t admits a left convolution kernel pt such that 


Ptf(h) = f* Pt (h) = f f {hg) p t (g) dg 
JG 

for all / e C£° (G). The function p t is called the hypoelliptic heat kernel of G. 

3.4. G ener alized Fourier transform. We will use the Fourier transform in Def¬ 
inition 2.11 and the Fourier inversion formula to find a spectral decomposition for 
the left-invariant vector field X corresponding to any X € g. 
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To describe a domain for the differential operator A, as well as for the Fourier 
transform of X , we turn to the notion of smooth and analytic vectors for a repre¬ 
sentation (due to E. Nelson [^7|, see also ]l5|]). 

Definition 3.7. Let 7r be a unitary representation of G in a complex Hilbert space 
TLtt , consider the map f v : G —> TL^ given by 

(3.3) f v ( g ) := tt ( g ) v for v £ H n . 

Then 

(1) v is an analytic vector if the map f v is analytic, and the space of such 
vectors is denoted by 

(2) v is C k vector, if the map f v is C k , and the space of such vectors is denoted 
by n k ; 

(3) Hff := th en v e is called a (G°°) smooth vector. 

It is well-known that Nf and Nff are dense linear subspaces of H n , and both are 
7T (G)-invariant ( || p. 230]). By |n], Theorem 3.3] the space Hff coincides with 
the Garding space of finite sums of vectors / (7r) v, f £ C£° (G), v £ 'Ht T - We will 
use the space because it works better with the GFT. In particular, we have 

that for 7r £ G 


(3.4) />)«£C, for all v £U™, f £ S {G), 

and if G is a connected, simply connected nilpotent group, then 


%—k 


). 


(3-5) HT-^j = S i 

where the first statement can be found in [||. Theorem A.2.7], the second in [|| 
Corollary 4.1.2] with J being the unitary isomorphism introduced in (2.12|). 


Definition 3.8. Let X £ q, it £ G, then define the differential operator dir (X) on 
with the domain 


V (dn (A)) := < v £ Hn : dn(X) ( v ) := Xf v = 


dt 


f v ( e tx ) exists ^ , 


t—0 


where f v ( e tx ) = tt (e tx ) v as defined by (3.3). 


Note that since tt is a unitary representation, by Stone’s theorem dir (A) is a 
closed, densely defined, essentially skew-adjoint operator on Moreover, as 

noted in || p.226] for any Afgwe have 


U™CV{dir (A)), 

(3.6) dir(X)(H™) QV(dn(X)), 

Xfv ( g) = fdn(x)(v) (g) j v s Wfff ,g £ G. 

Remark 3.9. Now we are ready to comment on Proposition 24]- Note that to 
make sense of all the ingredients in the statement, we need to know the domains 
of differential operators involved. If we start with as the domain of dir(X), we 
see that for a general unimodular group dn (A) o dn ( X ) is again a densely defined 
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second-order differential operator on Hf. Observe that this domain might depend 
of the representation 7 r which makes interpreting § difficult. 

There are some groups G for which this construction can be made rigorous. The 
first example is of the Heisenberg group H 3 . We refer to , |7^ . p. 65] for details, 
but the main point is that infinite-dimensional irreducible unitary representations 
n\ £ H 3 can be indexed by A £ R, and the Plancherel measure can be described as 
a measure on M. In this case TTf = and so is independent of X. We describe 
this group in Example 3. 1 . This construction can be extended to other nilpotent 


groups as well, and in particular we can use the fact that dn ( X) are differential 
operators with common domains if n £ G are realized as unitary representations on 
L 2 ( R n ~ k ,dx ). 

The next theorem can be viewed as a rigorous version of JlJ, Theorem 26]. In 
this case G is not assumed to be nilpotent. 


Theorem 3.10. For any X £ g, n £ G, v £ Tlf, f £ S (G), then 


X 2 f ( 7 r) v = d 2 n (X) / ( 71 ) v. 

Thus Anf(n) is a close, densely defined self-adjoint operator on H n with the do¬ 
main TTf such that for f £ S ( G ) 


(3.7) 


Ah/ (tt) = ( d27r ( X 0 I / O) ■ 


Proof. The proof is based on properties of the GFT that we formulated earlier. 
First, we observe that for operators / ( 7 r) and dn (X) by (T4) and ( |3.Ci| ) for f £ S (G) 
we have 


/(tt) (C)CCCH(di(I)), 
Finally, similarly to Js], p. 124] for f £ Cf (G) 


Xf(n) = 


dt 


t=0 J G 


f(ge tX )n(g ^ dg = 


d 

dt 


t =0 


[ /(y)’r(e tx g x ) dg = 
JG 


d 

dt 


t =0 


/ 7(5) 71 (e ‘ A )t r( 5 ^ dg 

JG 


-dn (X)f(n ), 


where this limit exists on V (dn (X)), and so in particular on by (|3.6| ). 

The result now follows since we can apply the same argument on the space of 
smooth vectors Hf. □ 


Using ([2.17) we can say that 


Ah: / HS (U^)dP(f) 


HS (H^)dP (0 


is an (essentially) self-adjoint operator for each n^ which acts by multiplication 
by the operator (X)I=i d 2 n^ ( Xi )) on the space of HS (H n i). If in addition G is a 
connected, simply connected nilpotent group, then for 7r = 7r/, m and f £ Cffi (G) 
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A#/ (7r^ m ) = 

can be described more explicitly, 
identified with an operator on L 2 ( 


XX ,r ‘. m ( x i) J /fa,m) 


By (p.5|) the operator cr7rz jtn (JQ) can be 


pn—fc 


, dec) which we denote by 


m 

(3.8) Ah (7Ti, m ) := X (fa) ■ 

i=l 

Now we can see that in the nilpotent case there is a natural semigroup corre¬ 
sponding to the GFT of the hypoelliptic Laplacian Ah (i q. m ), which in turn gives 
an explicit formula for the hypoelliptic heat kernel. Note that |I], Corollary 29] 
can be interpreted as a version of this formula (modulo the issues we mentioned 
previously), and in the nilpotent case we also refer to [^2[ Equation (6), p. 484], 


Theorem 3.11 (Hypoelliptic heat kernel). 

(3.9) Pt ( g )= f f X i(P2('Y{x)g))k l t (p 1 ('y(x)g),x)dx\Pf(l)\dl, 

Jungly JR n ~ k 

where k\ (x, y) is the heat kernel for a continuous semigroup with the generator 
Ah (^r/, m ) ■ This operator is a second order differential operator with polynomial 
coefficients on L 2 (R”~ fc ,dx) and with a nonnegative polynomial potential whose 
degree depends on the structure of the group G. 


Remark 3.12. By Remark 2.It we usually identify UGgf with an open subset of 
a Euclidean space and dl with the Lebesgue measure. Thus (3.9) gives a Euclidean 
integral formula for the hypoelliptic heat kernel on G. In addition, we can view 
U C\ gf as a non-commutative spectrum of a nilpotent group G. This goes back to 
an observation in J ^|j that no discrete spectra arises in this case, and so it is not 
surprising that we have an integral formula instead of a series as for a compact Lie 
group (e. g. jf/j. 


Pro of. Recall that the heat kernel pt is the convolution kernel defined in Definition 

m 

Ptf (g) = f*Pt{g)= I f ( gh)p t (h) dh. 

JG 

The heat kernel pt 6 S ( G ), and therefore by (2.18) for the representation 7p jtl 
on L 2 


yn—k 


h£L 2 (R n ~ k ,dx) 


, dx) there is an integral kernel kt £ S I 


pn —k 


pn—k 


) such that for 


(3.10) 

where 


(pt (7Tm) h) (x) 


h (y) k l ( x ,y)dy, 


(3.11) k\ (x, y) = J xi ( m ) Pt (7 (.x) 1 my (y)) dm, x,y el" k . 

Here we use Vergne’s polarization subalgebra m to define M. First observe that for 
any / G S (G) 
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Ptf (717, m ) = 7(7T/,m) ° ft (nl, m ) 

is a trace-class integral operator on L 2 (R n_fe , dx). To recover the heat kernel, we 
need to find a function / such that f (iri, m ) is close to the identity operator on 
T-U.m. This presents two problems: the first is that the identity operator is not a 
Hilbert-Schmidt operator, and the second is that in general ft (71 y m ) is only Hilbert- 
Schmidt, and therefore taking the trace in the Fourier inversion formula might be 
problematic. We will deal with the first issue by taking an approximate identity in 
G which is equivalent to defining the Dirac S function as a tempered distribution, 
and the second issue has been addressed in || as we explain below. 

Namely, let {<p n }^|L 1 be a bounded approximate identity, that is, <p n € C c (G ) 
be a sequence of functions such as in ]l2]. Proposition 2.42]. In particular, for any 
h G L 2 ( G) 


h * p> n 


L 2 (G) v 
- > n 

n—too 


and the operator norms of h 1 —> h * ip n are (uniformly) bounded. 

Now we can use the fact that by |^, Theorem 4.2.1] for any function in S ( G ) 
its Fourier transform is not just Hilbert-Schmidt, but trace-class. Therefore for 
the heat kernel pt we see that ft ( 7 r; :Itl ) is a trace-class operator, and because of 
the assumptions on the approximate identity {g>n}%Li we see that the convergence 
under the trace 


Tr (<p n (7T; im ) o f t (7T i>m )) -Tr (f t 

n—too 

holds. Now we can use the Fourier Inversion formula |l^, Theorem 7.44, p. 234] 
for nice enough functions h (such as pt*fn) on G 


h (g)= [ Tr(/i(7r 5 ) o ^(g))dP{(). 

Jg 

In particular, if h is in addition continuous, this is a pointwise identity. Applying 
this formula to Pt^Pm we have 


(PtVn) ( 9) = (Pt * Pn) (5) = 

[ Tr(pT*Pn(^) o 7 f(g))dP{£) = 

JG 

[ Tr(^ (tt ? ) of (tt 4 ) 07 f(g)) dP(£). 


In our setting the trace is taken in the representation space L 2 (R m ,da;), and by 
Theorem 2.15 we can identify G with equipped with the Plancherel measure 

| Pf(Z)|dJ, so we have the Fourier Inversion formula for Ptp n gives 
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(Pt<Pn) (g) = / Tr(P t (p n (7Ti !m ) o 7rj, m (p))| P£(0|tfl = 
Jung 5, 


'u ngj. 


Tr(<p>„ (7r iitn ) op t (7T; im ) o7r z , m (g))|Pf(Z)|dZ 


-»■ 


'ungt. 


Tr(pi (717, m ) o 7rz, m (g))| Pf(Z)|dZ = 


/ Tr(7r; im (g) op; (7T Z>m ))| Pf(Z)|cZZ, 

-Zc/nsj, 

where we used the fact that the convergence under Tr and therefore under the 
integral does not change after composing the unitary operator 717. m (g), and the last 
line uses the centrality of the operator trace. Now observe that if we have a unitary 
operator U on L 2 composed with a trace class integral operator 

(Kf) ( x) := f f (y) k (x, y ) dy, 

J R n - fc 

then U o K is a trace class integral operator with the kernel given by 

Uk (x,-), 

where U is applied to the kernel k (x, y) in the variable x. Then 


Tr(UoK)= / Uk(x,x)dx. 

J R n - fe 


Now we can refer to (3.10|) and (|3.11|) to see that 


(pi (n,m) h) (x) = h (y) k\ (x,y) dy, 

J R"- fc 

where kt is defined by ( Fill ). 

Recall that we identify the unitary representation 7 m with the representation 
U Um defined by (|2.13|), and so applying it to the kernel k\ in x we have 


(TRm ( 9 ) k\) (x, y) = xi {P 2 (7 (x) 9 )) k\ (pi (7 (x) g) ,y), 


and therefore 


(Pt<Pn) (g) - > 

n—>• 00 

f f Xi(P2h(x)g))k l t (p 1 (x{x)g),x)dx\Pi(l)\dl. 

Ju nsj, J R”- fc 

Now we recall that p t is a convolution kernel, and thus at least 

(Pt<Pn) ( g ) L (G) > Pt {g) ■ 

n—t 00 

Actually, this convergence is pointwise since the hypoclliptic heat kernel pt is con¬ 
tinuous and bounded. Thus 

Pt(g)=f f Xi{P2{l(x)g))k l t { Pl (^{x)g) ,x)dx\Pl{l)\dl. 

JungZ, Js. n ~ k 
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Finally we observe that k l t is the heat kernel for A h (in, m) on L 2 ( 'M . n ~ k , dx ) by [^2[ 
p. 481]. Note that this heat kernel corresponds to the initial point 7 -1 (x) 7 (y) as 
can be seen from (3.11). Recall that we can use Mal’cev basis which allows us to 
write left-invariant vector fields as first order differential operators with polynomial 
coefficients. By |^|, Theorem 4.1.1] in these coordinates d-K^ m (X,;) is a differential 
operator of degree 0 or 1 with polynomial coefficients with respect to (xi, X2,..., x n ), 
and therefore the operator A h is a second order differential operator with 

polynomial coefficients on L 2 (R n ~ k ,dx). □ 


Remark 3.13. The kernel k\ ( x,y ) is called the reduced heat kernel in 
among other papers, and it is the kernel of the following semigroup 


p. 481] 


ptAn (tt/ , m) 


/) (x) 


k\ (x, y ) / (y) dy, f e L 2 (R n k , dy) . 


Remark 3.14. Note that we can use some crude estimates for the heat kernel 
for the Schrodinger operator such as in ^ to prove heat kernel estimates for the 
hypoelliptic heat kernel pt. We use the following statement in ^tl\ . Proposition 
2.2.8]. For example, the reduced heat kernel k\ in the case when G is the Heisenberg 
group or the group G n introduced in Section 2.1 is the heat kernel for a Schrodinger 
operator C = — A + Vi on L 2 (l^, dx) with a nonnegative polynomial potential Vi. 
In this case Vi grows faster than |x|“ for some a, then there are positive constants 
Ci,Ci (depending on a etc ) such that the heat kernel for C satisfies 

h (x,y) < fWpI ex P (~ c ^ (M 1+ “ /2 + M 1+a/2 )) 
for all x, y € and 0 < t ^ 1. 

Example 3.1 (Heisenberg group). Let H 3 be the Heisenberg group identified with 
R 3 with the multiplication given by 

(a, b, c) • (a ', b' , c') := (a + a!, b + b', c + c' + — {ab' — ba')), 

and a Haar measure on then is the Lebesgue measure dadbdc on R 3 . Let 
{X , Y, Z} be the basis of the Lie algebra f) with the only non-zero bracket [X , Y] = 
Z. A polarizing (non-unique) sub-algebra for an element l = A Z* can be chosen as 
m = SpanjY, Z}. Thus the corresponding subgroup is 


M = expm = {exp(fcy + cZ), b,c€ 
identified with R 2 . The dual space of H 3 is given by 

H 3 = {tta, AeR] 

where 

7 Ta (a, b, c) : L 2 (R, C) —> 
f(x) 1—» 


Then for any / £ L 2 (R, C) 


L 2 (R,C) 

e 2ni\(c+xb+f) f ( x + a y 





24 


ASAAD AND GORDINA 


{dir x (X))f(x) = f (x), 

(dir a 00) / (x) = 2niXxf (x), 
(diT X (Z)) f (, x ) = 2iri\f (x). 


The Plancherel measure on H 3 is dP{ A) = |A|dA/47r 2 , where dA is the Lebesgue 
measure on R. By using the ingredients above we are able to define the Fourier 
transform f(X) of a function f £ L 2 (H 3 , C) as an operator on L 2 (R, C) 


(3.12) ^f(X)hj (x) = J f(a,b,c)(n x (a,b,c)h)(x)dadbdc 

= [ f(a,b,c)e 2mX (- c - xb+ ^h(x-a)dadbdc, h£L 2 ( R,C). 

J R3 


We can define a sub-Riemannian structure on H 3 by considering the two left invari¬ 
ant vector fields X = X(g),Y = Y(g), g £ H 3 . Then the horizontal distribution is 
given by 

H = Span{A, Y} 
and therefore the corresponding sub-Laplacian is 


A h = X 2 + Y 2 . 


The the Fourier transform A h defined by (T8) is 


A H (tta) / (x) = f" (x) - 4 tt 2 A 2 x 2 / (x) = f" (x) - V x (x) / (x). 

Note that by using the global coordinates (a, b, c) the sub-Laplacian is given by 

A H f(a,b,c)= (x 2 + F 2 )/(a,&,c) 



For x, y £ R we have 

7 _1 (x)m7(y) = exp(— xX) exp(bY + cZ) exp(yX) 


= (y 

The reduced heat kernel is then 


x,b,c— -(x + y)). 


h{x,y) = [ 7 1 {x)m r y(y))dm = [ e 2mXc p t {y - x,b, c - ^(x + y))dbdc. 

Jm Jr 2 z 

Observe that by (3.12) applied to pt and h being the Dirac 6 function (which can 
be made rigorous similarly to the proof of Theorem 3.11) we have 

(pt(X)S) (x) = / pt(a,b, c) (-rr x (a,b, c)5) (x)dadbdc 

J R 3 

= [ p t (a,b,c)e 2 ™ x (- c - xb+ ^6{x - a)dadbdc 

J R 3 

= f p t (x, 6, c)e 27r,A( ^ _c_ ^ ) dbdc 

J R 2 
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It is equal to the reduced heat kernel kt (0, x) and taking the derivative in t and 
using the fact that Y £ m, and therefore Y is a skew-symmetric on L 2 (M, dm), we 
see that 


d t k t (0,x) = A h (tt a ) k t (0,x) 




k t { 0 ,x). 


Finally, in this case we can make ( |L9| ) explicit. Namely, using the ingredients we 
described earlier, we see that the hypoelliptic heat kernel on H 3 is given by 


Pt{a,b,c) = f f e 27rli ( c + 2 )jd ( a -p x, x) dx\l\dl, 

RX R 

where k\ ( x , y) is the fundamental solution of Schrodinger’s equation with the gen¬ 
erator 


(■ Hif) ( x ) = -^2~ 12x2 /■ 

In this case this heat kernel can be found explicitly, namely, 


k l t (x, y) = 


l 


1/2 


2 ir sinh 


m) 


e s *( x,y \ where 


s\ (x, y) = l + y 2 ) coth (2 It) — 


xy 


2 sinh (2 It) 


Thus we can use this form for k\ to write an explicit expression for p t 


p t (a, b,c) = I I 21 cos ( 2 nl ( c + ^ ) ) k l t (a + x, x) dxdl. 


4. Hypoelliptic heat kernel on an jt-step Lie group 


I11 this section we define a sub-Riemannian structure on the n-step Lie group 
G n +i described in Section |2.l| , and then use the representations of G„+1 to find an 
explicit expression for the corresponding hypoelliptic kernel. 

In particular, we can use the matrix presentation (2.1) of the group G n +i. We 
now introduce the isomorphism (f> between G n +1 and R ra+1 by 


(t>(g) = (a, z), where z := {zx, 


0, 


where 2 is defined by (2.2). This isomorphism is a group isomorphism when R” +1 
is endowed with the following product 


(a,z) • (a',z') 


+ a', Zi + z[, 


n— 1 

■, z n +^2 

i —0 



Now, let us define a left-invariant sub-Riemannian structure on G n +1 as presented 
in Section ||. Consider two left invariant vector fields X\ , X 2 corresponding to 
X, Yj G Qn+i defined by ()l.l[), and let 


'H(g) ■= Span{X!(g),X 2 (g)}, 
Sg( x i(9),Xj{9)) = Sij- 
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Writing the group G n + 1 in coordinates (a, z) £ R” +1 , we have the following 
expression for the left-invariant vector fields Xi and X 2 

W A 
oa 

~ d d a 2 d a" -1 d 

2 <9zi a dz 2 2! 9z 3 (n — 1)! <9z n " 

The corresponding hypoelliptic Laplacian A h on G n +1 is given by 

A H f=(x 1 2 + X 2 2 )f, fee™ (G n+1 ). 

Our goal is to find an integral formula for the hypoelliptic heat kernel pt ( g ) as 
defined in Definition |3.6| where g is identified with a point in R n+1 . 

Notation 4.1. In what follows we denote by RJ _1 the set 

M"- 1 := {A„_i = (Ai, X n -i) £ M n_1 : K-i ± 0} . 

Theorem 4.2. The hypoelliptic heat kernel on the group G n +i is given by 

Pt{a, z) = 

2ni( Y2 AfcSfe(a;,z) | 

e V fe=1 / k t n_1 (x + a, x) dx\X n -i |dA„_i, 

R "- 1 R 

where k t " _1 (x,y) is the fundamental solution to a Schrodinger equation with a 
polynomial potential and 

k 

B k (x, z) := fjT~y xk ~^ k = !, 2 . ■ ■ •, n - 1, 

and 



is the Lebesgue measure. 


dA n -1 dX\...dX n -i 


Remark 4.3. Note that the integral should be taken overWf 1 , but as the integrand 
is 0 when A n _i = 0, we can instead integrate over R n_1 . 


Proof. This formula can be derived using Theorem 3.11. Recall that we described 


all unitary representations of G n .|_i in Section U, and now we can use these results 


to apply Theorem 3.11. For this purpose it is enough to consider representations 


in the support of the Plancherel measure P. Thus we identify 

G n +i = {TTAn-i : Aji-i £ R* -1 } , 

where 7 ta„_i is a unitary operator on L 2 (R, C) defined by 


7T An _ 1 : L 2 (R, C) —> L 2 (R, C), 

/ O) '—> (tt a — 1 (a, z)/) (x), 

27 Tli YZ 

(7TA„_!(a,z)/) (x) := e \ k=1 / / (x + a), 
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where polynomials Bk ( x ) are defined by ( 2.14 ). Recall that by ( 2.19 ) the Plancherel 
measure on G n +1 is identified with the following measure on R" -1 


dP(A„_i) = | Pf (l) \dm T (0 = |A n _i|dA n _i. 

Consider the representation 7rA n _! acting on the representation space L 2 (R, C). 


Using Definition T8 we consider dir\ ri _ 1 (Xi), i = 1,2 which are operators on 
L 2 ( R,(C), which can be found explicitly as follows. Let A„_i £ R" _1 , then 

d 


[dn a„_! [Xi)f] (x) = 


7T A n _ 1 (e tAl )/ (x) 


t =0 


(*, 0,..., 0)/ (a:) 


t =0 


f(t + x) = f ( X ) , 


t =0 


[dn A„_! (x 2 ) f] (x) = 


7i'A„_ 1 (e t - Y2 )/(a;) 


t=0 


TTA n _ 1 (0,t,...,0)f{x) 


t =0 


y.n—1 


— 27 vi I A n _i 




rO- 1 


(«-!)! fri (i-!) ! 


fix) 


Thus, under the GFT of the hypoelliptic Laplacian (as in Theorem ECU is 


® K- 1 ,a»- 2 ) i x ) = ii ~ 47r2 ( A «-i7uurnT + E 


y.n—1 


.7-1 


dx 2 


fa- 1 ) 1 (i-!) ! 


fix)- 


This is a Schrodinger operator with a polynomial potential 


£a„_! ;= ^ - U An _ 1 (x) / (x) 


where 


(4.1) 


VA._,W:= 4 nM^T7)i.-"- 


1 + U77^ J - 1 


7=1 


0 — !)■' 


Denote by k t ” _1 (x, ?/) the fundamental solution for the operator dt — £a„_i ■ Then 
applying Theorem 3.11, one gets the kernel of the hypoelliptic heat equation on 
G n+ 1 as follows 


Ptia,z) = 


//' 


271 -i \kB k (x) 


k^ n 1 (x + a,x) dx|A n _i|dA n _i. 
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5. Short-time behavior of the hypoelliptic heat kernel and the 
Trotter product formula 

In this section we recall the techniques used by Seguin and Mansouri in pof , 
where they show how the Trotter product formula for perturbation of a semigroup 
can be combined with an explicit formula for the hypoelliptic heat kernel to study 
the short-time behaviour of this heat kernel. 

The following version of the Trotter product formula is well-suited for our pur¬ 
poses. Suppose C is an operator that can be written as a sum of two operators 
C = A + B. Then we can relate the semigroups generated by A and B with the 
semigroup generated by C as follows (see 111, Corollary 5.8]). 


Theorem 5.1 (Trotter product formula). Let (T t ) t ^q and (S t )t^o be strongly con¬ 
tinuous semigroups on a Banach space X satisfying the stability condition 

\\[Tt/NS t / n } N \\ < Me wt , for allt^0,N€N 

for some constants Consider the sum A + B on D := D{A) D D(B) 

of the generators (A,.D(A)) of (T t )t ^o and (B,D(B)) of {St) o> an d assume that 
D and (Ao — A — B)D are dense in X for some \q ^ w. Then the closure of the 
sum of these two operators C := A + B generates a strongly continuous semigroup 
(£4)t>o given by the Trotter product formula 


(5.1) 


U t x= lim [T t/N S t/N } N x 

N—too 


with uniform convergence for t on compact intervals. 

As Seguin and Mansouri observed, in most of cases the expression [T{t/N)S{t/N)] N 
is too complicated to be described explicitly. But in the special case when the oper¬ 
ators satisfy some additional conditions and the Banach space X is a nice function 
space such as L p , ^ p < oo, (see 30, Section 3.3] for details), ( ^d| ) takes the form 


U t f = T t f + lim — 
J J N^>oo N 


rriN — k nmfe 
/ , 1 t/N nj t/N 


f + o(t 2 )f , 


where 0{t 2 ) is an operator D t acting on / such that H-Dj/H^i < Mf 2 ||/|| L i for a 
constant M and for all t small enough. 

Note that their main result J30|, Theorem 1] uses a number of assumptions for¬ 
mulated on (30[ p.3904]. Some of them are ambiguous (such as Assumption 1 which 
does not address the issue of the domains of unbounded operators), therefore we 
would like to use more concrete decompositions we have as a result of using Kirillov’s 
orbit method. Namely, we consider the semigroup with the generator Ah on 

L 2 (R n ~ k ,dx) defined by (T8). That is, we would like to write A h (7U,m) as a sum 
A 1 + B l which are generators of the semigroups {T \} t>0 and {<St} t>0 respectively, 
and which satisfy the following assumptions similar to |io|. Proposition 3]. Note 
that they base their analysis on the terminology of |I|] which in particular leads to 
potential issues with ]30|, Theorem 1]. But in our setting the direct integral termi¬ 
nology is not necessary, and in particular, all operators and their semigroups are 
defined on L 2 (JS . n ~ k , dx) , that is, on a space which does not depend on 7r^ m . That 
is, in the case of nilpotent groups dependence on elements in the unitary dual G is 
much easier to track. 





HYPOELLIPTIC HEAT KERNELS ON NILPOTENT LIE GROUPS 


29 


(1) For each t ^ 0, Tj is an integral operator with kernel h l t (x,y) 

i T tf)( x )=[ h l t{x,y)f(y)dy, 

(2) There exists an integrable function H^ N (x, y). uniformly bounded in x by 
an integrable function G l t (x,y) for all N ^ 1, such that 

(t/N) (JtI /n ) N B 1 + ...+T l t/N B l {T l t/N ) N ^ f(x) = J H l t/N (x, y)f(y)dy. 

Then |3(j, Theorem 1] can be interpreted as follows. The operator A h is the 

generator of the semigroup e tA H(7u, ro ) w } 1 i c i 1 j s an integral operator on L 2 (W l ~ k , dx) 
which can be written as the following expression for all small enough t 


(5.2) (e tAa ^ 1 ’ m )/) {x) = 

J (ji[(x,y)+t lim H l t/N (x,y))f(y)dy+{o(t 3 / 2 )f ) (a;), 

where the integral kernels h\ and Hi are defined above, and as before 0 (t 3 ^ 2 )f is 
an operator D t acting on /, such that ||-D t /||£i ^ Mt 2 ||/||ii for a constant M and 
for all small enough t. Combining this with the reduced heat kernel introduced in 
Remark 3T3, we see that 

k\(x,y) =h\{x,y)+t lim H l t/N {x, y) + 0{t 2 ). 

N —>oo ' 


Recall that for the Heisenberg group H 3 and the group G n +1 


Ah (tt;, m) = A — Vi, 

where Vi is a positive polynomial potential on L 2 (R, dx) . Let 

(A l f) (x) := A f(x) = f"(x) 
and B l be the multiplication operator 

(B l f) (x) :=V l (x)f(x). 

We can write V( (x) as 

Vl \ x ) — — \ a 2m X + a 2m-l X + ' * * + a 2 X + a l x + a 0j j 

where m is the degree of the polynomial Vi (x), and a l 2m and a l 0 are positive con¬ 
stants. Then jjol Corollary 2] is applicable, the equation (5.2) becomes 


(5.3) (e tAH(7ri ' m) /) (x) 


x = 


1 (*-v ) 2 

e 4t 


2m 


\[\t rt 


i - a l 0 t - 1 £ E xl y k ~ l + °(* 3/2 )/(*) 

^ fc=1 K 1 »=0 / 


for / € C£°(R). It is clear that for more general Schrodinger-like operators in The¬ 


orem 3.9 this approach can be used as well without making additional assumptions. 
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5.1. Application to the n-step nilpotent Lie group G n + 1 - Recall that for 
G ra _|_i the operator A h is 


(7r,, m )) / (a:) = /" (a:) -Vi(x)f{x),f€ C c (G n+1 ), 
where by © the potential is given by 


Va„_i (x) = 4?r 2 


An —1 


(«-!)! 

We will use the following notation 




Afc := (Ai, A2,..., Afc). 
Then as we mentioned previously 
A A ^f(x) = f" (x), 


(x) = - \ r , 




d-i 


(n —!)! ^ O'-l)! 


/(*) ■ 


Therefore the coefficients of the polynomial potential are 




fc+1 


1 := Y AAfc+ 2 -i, fc = 0,1,..., 2(n - 1). 


2=1 


Finally, we use ( J5.3| ) to see that 
(e An " 1 /) (x) = 


/ 


t ~ 

Jr \/47rt 


e “ 


fc+i 

2(n-i) S A(Afc+2-I fe 


1-A \t-t y 


1=1 


\ 


k + 1 

fc=l 2—0 


E*V-‘ 


f(y)dy 


+ (o(< 3/2 )/) (*) 

with the reduced heat kernel being 

fc t An_1 (x, y ) = 


1 _ Q-y ) 2 

e 4t 


•\/47rt 


2(n 1) \/c—(— 1 \ \ k 

!-❖-* e E, - i t ; r~‘ E*y-‘ 


fc=l 


0(f 3/2 ) 


2=0 


1 _ (a-y) 2 

e 4t 




2(n 1) >/c—|— 1 \ \ & 

E fc+i 


fe=i 


2=0 


where we used that ©(r 1 / 2 )^ 2 ) = 0(f 3 / 2 ). 
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Now we can use Theorem [h2| to see that the hypoelliptic kernel is given by 
Pt(a, z) = 


2ni[ ^2 \ k B k (x,z) 

k=i 


|A„_i|fc f An 1 (x + a,x)dxdA n -i, 


therefore 


Pt(a, z) = 


y/Afft 


27T i\ ^ kB k 

. k =1 


fc+1 

2(n—1) X] ^Afc+2-Z k 

y, ^TTi —i>+«)***-*+o(* 2 ) 

k=l + i=0 


dx|A„_i|dA„_i. 

Denote by P 2 ( ra _i) the polynomial of degree 2(n — 1) 

2(n-l) T-^k+l . , fc 

P 2 (n-i)0r,a) :=A 1+ )T ^ 1=1 , * fc+2 ~* + a)^ 

fc=l i =0 

Choosing the substitution 

1 - £P 2 (n-!)(*, «) = e~ tp ^-^ + <D(t 2 ), 

we get the following expression 


Pt(a,z) = 


y/fart 


e 44 x 


2ffi( Z AfcBfe) 
e fe=1 


( e -^Cn-p(-,«) + 0( i 2 )) d,x\\ n _\\d\ n _idA n _ 2 = 

n — 1 

n 2 r 2iri('y' A k B k ) , . 

—7==e _ 47 / e *= 4 e- tP2 ("- 1 )("’ a ^x|A n _ 1 |dA„_ 1 dA n _ 2 

V 47rt Jp 


1 


a —1 

27ri( Z ^ kB k ) 


+ 7=— e 44 / e fc=1 0(r)da:|A n _i|dA n _idA n _2. 

V 47rt Jr" 

The first part is a C 00 function, and one can use special functions to find a more 
explicit form of this integral. The second integral is also a C°° function which is of 
order 0(t 3 / 2 ). 
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